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4.0 Abstract 


One focus area of the National Aeronautics and Space Administration (NASA) is to improve 
aviation safety. Runway safety is one such thrust of investigation and research. The two primary 
components of this runway safety research are in runway incursion (RI) and runway excursion 
(RE) events. These are adverse ground-based aviation incidents that endanger crew, passengers, 
aircraft and perhaps other nearby people or property. A runway incursion is the incorrect presence 
of an aircraft, vehicle or person on the protected area of a surface designated for the landing and 
take-off of aircraft; one class of RI events simultaneously involves two aircraft, such as one aircraft 
incorrectly landing on a runway while another aircraft is taking off from the same runway. A 
runway excursion is an incident involving only a single aircraft defined as a veer-off or overrun 
off the runway surface. 

Within the scope of this effort at NASA Langley Research Center (LaRC), generic RI, RE and 
combined (RI plus RE, or RUNSAFE) event models have each been developed and implemented 
as a Bayesian Belief Network (BBN). Descriptions of runway safety issues from the literature 
searches have been used to develop the BBN models. Numerous considerations surrounding the 
process of developing the event models have been documented in this report. The event models 
were then thoroughly reviewed by a Subject Matter Expert (SME) panel through multiple 
knowledge elicitation sessions. Numerous improvements to the model structure (definitions, node 
names, node states and the connecting link topology) were made by the SME panel. Sample 
executions of the final RUNSAFE model have been presented herein for baseline and worst-case 
scenarios. Finally, a parameter sensitivity analysis for a given scenario was performed to show the 
risk drivers. 

The NASA and LaRC research in runway safety event modeling through the use of BBN 
technology is important for several reasons. These include: 1) providing a means to clearly 
understand the cause and effect patterns leading to safety issues, incidents and accidents, 2) 
enabling the prioritization of specialty areas needing more attention to improve aviation safety, 
and 3) enabling the identification of gaps within NASA’s Aviation Safety funding portfolio. 
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5.0 Introduction 


One focus area of the National Aeronautics and Space Administration (NASA), enabled 
through the former Aviation Safety Program (AvSP), now the Airspace Operations and Safety 
Program (AOSP) 1 , of the NASA Aeronautics Research Mission Directorate (ARMD), and in 
cooperation with the Federal Aviation Administration (FAA), is to improve aviation safety. 
Specifically, this Program seeks to provide increasing capabilities to: 

• predict and prevent safety issues; 

• monitor for safety issues in-flight and lessen their impact should they occur; 

• analyze and design safety issues out of complex system behaviors; 

• analyze designs and operational data for potential hazards. 

The AvSP / AOSP explores hardware and software systems (technologies or products) that 
will operate in the Next Generation Air Transportation System (NextGEN) 2 . Runway safety is one 
thrust of investigation and research. The two primary components of runway safety are runway 
incursion (RI) and runway excursion (RE) events. 

A runway incursion is the incorrect presence of an aircraft, vehicle or person on the protected 
area of a surface designated for the landing and take-off of aircraft (as defined by the FAA Office 
of Runway Safety) 3-4 . A runway excursion is an incident involving only a single aircraft defined 
as a veer-off or overrun off the runway surface 4 . In short, RI and RE events are adverse ground- 
based aviation incidents that endanger crew, passengers, aircraft and perhaps other nearby people 
or property. Additional detail about RI and RE events is provided in subsequent sections of this 
document. 

Within the scope of this effort, statistical RI framework 5 , and generic RI, RE and combined 
(RI plus RE, or RUNSAFE) event models 6-7 have each been developed and implemented as a 
Bayesian Belief Network (BBN) 8 ' 10 . Data from the FAA and descriptions of issues from the 
literature searches, described subsequently, have been used to develop the BBN models. The 
development of the RUNSAFE (combined RI and RE) BBN model 7 , and some sample executions, 
are documented later in this report. Other similar BBN modeling efforts have been recently 
documented within a group working at NASA Langley Research Center (LaRC) 8 - 11-13 . More 
discussion about BBN models also follows subsequently. 
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The NASA Aviation Safety Program conducts cutting-edge research to produce innovative 
concepts, tools, and technologies that can improve the intrinsic safety attributes of current and 
future aircraft. The AvSP research centers have defined a set of Technical Challenge (TC) 1 issues 
that are aligned with program goals and project objectives. These TC issues serve to focus research 
toward solving aviation safety problems and provide a consistent framework to focus, direct, plan, 
execute, manage, and communicate Center-distributed research. Among the TC issues relevant to 
this work are: 

• Assurance of Flight Critical Systems (air traffic control operations) 

• Discovery of Precursors to Safety Issues 

• Assuring Safe Human-Systems Integration 

• Improve Crew Decision-Making and Response in Complex Situations 

The NASA and LaRC research in event modeling through the use of BBN technology is 
important for several reasons. These reasons include: 1) a means to provide a clear understanding 
of the cause and effect patterns leading to safety issues, incidents and accidents, 2) to enable the 
prioritization of specialty areas needing more attention to improve aviation safety, and 3) to enable 
the identification of gaps within NASA’s Aviation Safety funding portfolio. 

5.1 Runway Incursion Events 

Again, a runway incursion is the incorrect presence of an aircraft (AC) , vehicle (VEH) or 
person (pedestrian or PEP) on the protected area of a surface designated for the landing and take- 
off of aircraft (as defined by the FAA Office of Runway Safety 34 ). Generally, RI events are 
reported by air traffic control (ATC) personnel (this phrase is intended to include those personnel 
that may be controlling only ground traffic at airports) in one category (Cat) among several severity 
categories, originally defined as: 

• Cat A = an accident or near miss occurred 

• Cat B = significant potential for collision existed 

• Cat C = ample time and/or distance existed to avoid a collision 

• Cat D = an RI event with no immediate safety consequences 

• Other = RI events that have not yet been properly classified 


3 



The FAA has recently revised these severity categories to be: Accident, Cat A (Near Miss), Cat 
B, Cat C, Cat D and Cat E (Other) as noted above 3 ' 4 . Once initiated, RI events are short in duration 
and timing is critical; typical landing and takeoff times are 20 to 30 seconds and the event severity 
can easily escalate with just slightly different timing. 

As shown in the timeline of Figure 1, an RI event consists of a development phase (blue box) 
and a mitigation phase (red). During the development phase, various circumstances (bad weather, 
poor airport layout, confusing communication with the ATC, etc.) contribute to the occurrence and 
possible severity of a future RI event; the development phase may take place over minutes or 
hours. During the mitigation phase, various actions may take place that can reduce the severity of 
the RI event; this phase may only be a few seconds. Three intermediate instances in time as marked 
in Fig. 1 are also relevant: 1) at some point in time, one or more of the people involved may realize 
that anRIis imminent but has not actually occurred yet, 2) at some point in time, due to the physics 
of the situation, the RI event becomes unavoidable, and 3) at some point in time, the RI event is 
actually initiated. Under some circumstances, the three instances in time (indicated as 1, 2 and 3 
in Figure 1) coalesce and there is no overlap between the development and mitigation phases. 



Development 

Phase 


Figure 1. Runway Incursion timeline sketch. 

One can imagine a situation where a pilot, on final approach for a landing, observes a VEH 
being driven at high speed toward the runway on which their aircraft has been cleared to land. An 
RI event has not yet been initiated but the pilot suspects that an RI event will occur. At this point 
(instance 1 in Figure 1), the pilot has ample time to initiate a go-around (whether self-initiated or 
directed by the ATC); the VEH may stop before entering the runway, and the RI event may never 
technically occur. Under slightly different circumstances, the pilot may only realize that an RI 
event will occur after it is too late to be avoided (i.e., the VEH will enter the runway without 
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authorization, instance 2 in Figure 1) and begin to initiate a go-around that results in a near miss 
with the VEH; without the go-around attempted, this event may have resulted an AC / VEH 
collision. Again, under slightly different circumstances, the pilot may only realize that anRI event 
has occurred after the VEH enters the runway without authorization (RI event initiated, instance 3 
in Figure 1); any attempted mitigation at this point could still result in an AC / VEH collision, a 
near miss, or the AC may have an accident on its own attempting to avoid the VEH. There are 
many complex situations to consider in this context. 

In reviewing the literature on this topic, a recent NASA study on non-towered airports 14 
indicated that the number of RI events is increasing with time, with about half of the events being 
of low severity and the remainder being split among moderate, high, and severe RI events; among 
these events, intersecting runaways are noted as the highest contributing factor. A 2013 
presentation by the Boeing Company 15 shows that flight hours, departures, and the size of the 
worldwide fleet have generally increased, while accident rates have remained essentially flat (but 
at a very low level) over the last 20 years; the same presentation points to about 61% of all fatal 
accidents and about 50% of onboard fatalities worldwide being associated with final approach, 
landing, takeoff and initial climb during the period 2004 through 2013. A recent U.S. Department 
of Transportation, Volpe Center 16 report shows that the spacing of parallel runways has just a small 
effect (if any) on the number of RI events across all severity categories; the same reports illustrates 
that crossing the hold short line, entering the runway and crossing a runway as the most likely 
types of RI events. A recent journal article 17 illustrates a dramatic increase in the number of RI 
reported in 2008 compared to previous years, with pilot deviations always being the largest source 
of these events. A recent FAA report 18 described the strong correlation among airport geometry, 
complexity and various communication tools (including signage and runway markings) with RI 
events. A Pilots Association report 19 illustrates an increase in RI events with air traffic, but with 
overall the RI event rate being less than 6 per million operations; this reports also points to major 
domestic airports (Chicago, Atlanta, Dallas/Fort Worth, Los Angeles, St. Louis and Philadelphia) 
as having the greatest number of RI events. 

With the goal of improving runway safety, a statistical analysis 5 of the Runway Incursion (RI) 
Database 20 from the FAA Safety Information Analysis and Sharing (ASIAS) website 21 and the 
FAA Air Traffic Activity Data System (ATADS) 22 , also from the FAA, was conducted to ascertain 
its relevance to the top ten challenges of AvSP. The information contained in the RI database was 
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found to contain data that may be relevant to several of the AvSP top ten challenges 1 including: 1) 
the assurance of flight critical systems [i.e., airport operations], (2) the discovery of precursors to 
safety issues, and 3) improve crew decision-making and response in complex situations. Prior 
conference papers by this author fully documented the statistical analysis of RI data 5 and the initial 
development of a BBN model for RI events 6 . A subsequent conference paper 7 extended this BBN 
model to also include RE events; these three reports together serve as the basis for much of what 
is reported in this NASA TM. 

Some of the important findings of the statistical analysis 5 were that: 

• while the number of RI events was found to be increasing over time, there has been 
essentially no change in the number of higher severity RI events over the time period 
examined (2001-2011) 

• the assumed risk level was found to be increasing over time 

• part of the observed increase in the number of RI Events and the assumed risk level can be 
attributed to an RI definition change by the FAA between 2007 and 2008, described in a 
2010 paper by Chapman 23 

• Chapman also notes that pilots consistently rated RI events at higher severity than the FAA 
controllers that typically report the RI events 23 

• while a few airports, such as Winston / Salem, NC and Fort Wayne, IN had a number of 
RI events well above the mean value for all the airports considered, those airports with 
large traffic volume, such as O’Hare, Chicago, IF and Atlanta, GA clearly stand out with 
statistically significant high average risk sums 

• when the average risk sum is normalized by the air traffic volume associated with each 
airport, other smaller airports stand out with statistically significant high risk levels , 
meaning that a flyer’s actual risk of being involved in a RI event at high traffic volume 
airports may actually be significantly lower than the risk at other smaller volume airports 

• by far the most prevalent cause of RI events is pilot error (about 72% of all RI events) 

• an unauthorized person or vehicle account for about 19% of the RI events 

• among the pilot errors, two contributing factors were readily identified as major 
contributors: accidental use of the wrong runway or taxiway (about 25% of pilot errors), 
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and confusion about the extent of authority granted to the pilot at a specific time by the air 
/ ground / local traffic control personnel (about 20% of the pilot errors) 

• among weather factors, adverse lighting conditions may have the greatest overall and the 
most consistent contribution to severe runway events 

• snow (and other freezing conditions) are less of a potential contributor to runway events 
(only 2% to 9% of RI events cite these conditions) than wind, rain, visibility and lighting 

• surprisingly, only about 16% of the RI events examined included some form of 
intervention, where a corrective or mitigative action was taken 

• when an intervention or mitigation occurred, these actions were successful in reducing the 
RI event severity about 70% of the time 

• a “go-around” issued to incoming planes was the most common form of intervention 

5.2 Runway Excursion Events 

The RE event rate is quoted in several references: about 1 to 2 per mill ion flights for the period 
1990 through 2006 24 , up to 16 accidents and incidents per year during the period 1978 to 2008, 
and 30 runway excursion accidents per year 25 . The approach and landing phases of flight have 
shown little improvement in safety over the last decade (up to 2008) and RE events are the third 
greatest source of aircraft crashes behind in flight loss of control and controlled flight into terrain 26 ; 
according to this source, the frequency of runway incursions is about half that of runway 
excursions, which may amount to 10 to 20 overruns and veer-offs each year 26 ; the data used in this 
report suggests that severe RI (Category A / B) are together about as common as RE. If the reader 
is concerned that the references cited here are out of date, prior versions of the 2013 Boeing 
report 15 , dating back to 2010 (and considering data back to 2004) show almost identical numbers 
for RI end RE events over a broad number of years. 

According to one source 24 , landing RE events are the most common, representing about 77% 
of all RE events. Some contributing factors are shared across the various types of RE events. The 
most common contributing factor associated with landing overruns is wet/contaminated runways, 
with long landings being the second most common contributor. Several other contributing factors 
are also noted (incorrect decision to land, speed to high, late/incorrect use of brakes, late/incorrect 
use of thrust reversers, aquaplaning, tailwind and being too high on approach). For landing veer- 
offs, the most common contributing factors include crosswind, nose wheel steering problems, 
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collapsed landing gear, hard landing, tire failures and asymmetric power. The most common 
contributing factors to takeoff overruns include late abort/reject and an inaccurate estimation for 
takeoff mass. The most common contributing factors for takeoff veer-offs include inadequate 
supervision of the flight 24 . 

Other additional causal and contributing factors for RE events exist, which are summarized 
here and noted again later in this report where they are particularly relevant. While the direct role 
of air traffic control (ATC) personnel in runway excursions was relatively small 24 , the ATC 
personnel may contribute to RE events by providing poor, complex or incomplete instructions. 
Several other contributing factors, including communication/coordination/planning, poor decision 
making processes about landing or takeoff under adverse circumstances and approach/takeoff 
procedures are important 27 . One contributing factor not previously mentioned is the inconsistent 
reporting of runway conditions and braking action at airports across the world 26 . Numerous 
challenges exist for improving runway safety for existing airports 28 because, due to fixed runway 
layouts and surrounding populated areas, these facilities may lack the flexibility to implement 
recommended runway safety mitigation strategies. While contaminated runways (ice, snow, slush, 
wet or flooded) are a significant contributor to RE events, almost 47% of RE events occurred on 
dry runways 29 . Takeoff runway excursions were likewise predisposed by a number of factors 30 . 

Some other reports discuss other model development and application efforts, also aimed at 
improving runway safety. For instance, one reference describes an analysis tool to quantify risk, 
support planning, and engineering decisions when determining runway safety area requirements 
for various types and sizes of airports 27 . The FAA is developing integrated risk models to forecast 
the risk and assess the impact of additional control measures at specific airports based on traffic 
volumes, complexity, and environmental factors 31 . Another study is taking a more novel and 
holistic approach to make sure that resources spent by airports to improve runway safety are 
actually used to address the most common types of RE events 32 . Yet another study employed 
human- in- the -loop simulations to evaluate traffic capacity at the Los Angeles International 
Airport 33 . An automated risk rating model for RI events was presented in another report 34 . Another 
document by the same author explores what is known about the human errors and other factors 
that have been identified as contributing to runway incursions, and offers some error mitigation 
strategies 16 . An example of a BBN devoted to RI events is given in another source 35 . 
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5.3 Bayesian Belief Network Models 

The modeling philosophy includes the use of a generic, high-level, system- integrated modeling 
with a systems level risk-based causal model. It should capture the multi-dependencies 
(interactions) of causal and contributing factors from various problem domains. However, the 
modeling should not be a representation of a specific accident/incident case, nor a detailed 
simulation analysis. 

In general, the modeling steps undertaken include: 1) determining the causalities and cause -to- 
effect relations based on the historical risks and anticipated future risks from safety data/database 
and literature reviews, 2) constructing a baseline risk-based causal model as a BBN, 3) conducting 
Subject Matter Experts (SME) Knowledge Elicitation sessions to review the baseline model 
structure and to elicit the Conditional Probability Table (CPT) values for the baseline model 
without product insertions, and 4) inserting the NASA safety technologies/products into the model 
and eliciting CPT values with products included. The expected modeling results include 1) a 
quantification of the relative likelihood of concerned aviation risks, with technology products 
inserted and without, 2) an assessment of the direct risk mitigation effectiveness of the NASA 
safety technologies/products, 3) a portfolio gap analysis and 4) a sensitivity analysis for risk 
drivers. 

Several recent BBN modeling efforts 6 ' 7 - 9 ' 13 have been undertaken to support the AvSP portfolio 
assessments and to determine if the AvSP technologies are addressing/mitigating aviation safety 
problems. The characteristics of issues selected for modeling are: 

• A significant accident category based on the historical data and/or future trend 

• Alignment with the focus and research areas of AvSP 

• Broad coverage on AvSP safety technology products 

• Many underlying causal/contributing factors that lead to aviation accidents 

• Suitability for a high-level system analysis and modeling 

A typical BBN consists of the model structure and the model content. The model structure 
consists of a set of relevant definitions, as well as the node names, the node states, the ordering of 
the defined states for each node to facilitate SME comment, the connecting link topology and the 
connecting link priority as they enter specific nodes (again, to facilitate SME comment). The 
model content consists of the sets of marginal and Conditional Probability Table values. During 
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the first phase of a typical BBN development cycle, NASA researchers develop (based upon 
database and literature search) and propose a model structure to an SME panel; the development 
step may take months to complete. Then, the various elements of the proposed structure are 
reviewed, modified and validated by the SME panel. Once the model structure has been agreed 
upon and validated by the SME panel, aCPT elicitation process (model population) is conducted 
by a facilitator on behalf of NASA to determine the appropriate model content. Some portions of 
the model review, modification, validation and population can be conducted in parallel. Once the 
model has been populated, it is executed to obtain a set of baseline results and a baseline sensitivity 
analysis. 

Although both RI and RE events may involve numerous contributing factors (e.g., airport 
layout, airport operations, weather, training and mechanical failures), RI events are more complex 
than RE events. RI events are “people intensive”, possibly involving multiple pilots, controllers, 
airport employees or contractors and perhaps other participants. There are also organizations 
(FAA, Airport Management, Aircraft Operators, etc.) that support each of the people directly 
involved in the runway safety events. RI events are also “communication (Comm) intensive”; 
several instances of two-party communications must simultaneously function properly in order to 
avoid problems. Two-party communications involve both the content and hardware transmission 
of information. Instances of two-party communications exist between all the people involved in 
the event (e.g., pilot to pilot, pilot to controller, controller to controller, controller to airport 
personnel). All runway safety events are short in duration and timing is critical; typical landing or 
takeoff reaction times are about 20 seconds (or less) and the event severity can easily escalate with 
just slightly different timing. The RI and RE models could be joined together through a set of 
common definitions for accident and incidents, based upon the level of aircraft damage and 
passenger injuries 36 but this was deemed out of scope for the current models. 

An SME panel consisting of four consultants was assembled to review the model structure and 
to populate its content. The SME panel included two pilots and two other aviation expert 
consultants. The preliminary BBN RI event model discussed in the prior conference paper 7 
(describing the SME session of November 2013) was substantially modified and simplified during 
the second SME panel review (April 2014), as shown in Figure 2. The nodes in Figure 2 are color 
coded to indicate associations among the various groups. Generally, the flow of specific 
contributing factors through causal paths is from left to right in the figure. The SME panel 
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validated many of the proposed definitions and most of the proposed model structure. However, 
the SME panel also provided significant clarification of several essential definitions within the RI 
event model. It is intended that the node names are suggestive of the states of each node; hence, 
limited clarifying information is presented about each of the nodes. Most of the nodes in the RI 
event model are binary, meaning they have only two possible states: the issue is present, or not, in 
the RI event under consideration. Where more than two states are present in a node, this will be 
made clear from the explanation of the node given subsequently. The goal of the SME elicitation 
is to provide probabilities for each of the possible states; for example, for the node “Airport 
Layout”, the SME goal is to determine the probability that the Airport Layout is an issue or not in 
the RI event. 

The SME panel unanimously agreed that developing a model for runway safety was much 
more difficult than developing one for in-flight operations. Two members of this SME panel have 
also participated in prior similar model reviews hosted by this NASA team for different 
applications. However, in this case, it was quite challenging to even achieve agreement on the 
basic structure of the BBN model among the NASA team and the SME panel. Numerous 
alternative models have been developed, discussed and discarded by the NASA team, either 
because they did not provide a satisfactory causal path, or because they were deemed to be too 
complex for use within the SME elicitation process for the purpose of portfolio assessment. The 
model proposed and discussed during the November 2013 session was significantly changed by 
the SME panel at that time and then significantly changed again by the same SME panel (April 
2014). Yet another SME review by the same panel (July 2104) altered the model structure further. 
However, through all the discussions and modifications , the runway safety model has been steadily 
improved and clarified. 

5.4 Methodology and Software 

The scope of the work detailed in this document employed three commercially-distributed 
software products: Microsoft Excel 37 and Design- Expert (versions 8 and 9, referred to herein as 
DX8 and DX9, respectively) from Stat-Ease, Inc 38 and the Hugin Explorer software (version 8.1) 
from Hugin Expert A/S 39 . The first two pieces of software (Microsoft Excel 37 and Design- 
Expert 38 ) were generally used during the data collection and analysis phase of the work 5 , while the 
Hugin Explorer software was generally used during the BBN modeling phase of the work 6 ’ 1 . The 
general workflow that was employed in this study was first to download the RI data set from the 
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ASIAS web site. Then, the air traffic volume data set 20 was downloaded from the FAA ATADS 
web site 22 . These datasets were downloaded in Microsoft Excel 37 format and this software was 
used to sort and extract the information of interest in addition to some statistical processing. The 
intent of this data pre-processing was to develop representative marginal and conditional 
probabilities for use in the BBN modeling of specific events, causes, combinations of contributing 
factors, and the participant types (aircraft classes, and if vehicles or pedestrians were involved) of 
RI events that occurred. In this context, it is not necessary that these searches and sorts be 100% 
accurate, but merely that they provide reasonable guidance about the relative percentages. Having 
prepared the data set into suitable formats, the data was then imported into DX8 for the 
development of response surface (RS) models via the analysis of variance (ANOVA) technique, 
and for additional statistical processing with the software 40-44 . Having identified primary and 
secondary causes and contributing factors, a series of increasingly more detailed BBN models were 
developed with the Hugin software and discussed among the team to determine which represented 
the best way to model the RI and RE event structures. These RI and RE event models were then 
present to, and discussed at length with, the SME panel; the SME panel made many additional 
clarifications, simplifications and structural changes to the BBN models. 

The software choices noted above simply represent software currently available to the author, 
and software packages to which the author is quite familiar, but in no way represent an official 
federal government or NASA endorsement of these software packages. However, these software 
packages are known to include the desired capabilities for accomplishing the objectives of this 
study. 

The original RI database that was used consists of 10459 records for RI events from October 1, 
2001 through September 30, 2011. The structure and use of this data set has been previously 
documented in detail 5 - 20 . 

The primary Federal Air Regulations (FAR) aircraft categories of interest within this modeling 
effort are 121 (commercial), 135 (air taxi) and 91 (general aviation), but other categories of aircraft 
were included in the data set. In order to compare RI incident rates at various airports, the event 
data was combined with a data set of aircraft traffic volume. The air traffic volume data set 
provided quantitative measures of how many landings and takeoff (grouped together) occurred by 
year at each of over 400 domestic airports in several categories of aircraft. The total air traffic 
volume for each airport is also provided. These datasets were used together to investigate issues 
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such as the percentage of the air traffic volume (total, or at a specific airport) that resulted in 
runway incursions over a given period of time, and the true, traffic-normalized risk level that is 
associated with those RI events. Again, the intent of the various data analysis operations was to 
develop representative marginal and conditional probabilities of specific events, causes, 
combinations of contributing factors, and participant the types of RI events that occurred. The data 
operations need only provide reasonable guidance about the relative percentages. All the 
quantitative data was then used as the basis for developing the BBN models. 

6.0 Model Development 

6.1 Runway Incursion Model 

Not all possible combinations of these objects (AC, VEH and PED) are of interest to NASA, 
e.g., VEH in combination with VEH / PED is not a subject of this study. To avoid ambiguity for 
the most important Cat A events, the RI event severity rankings used henceforth in this report are: 
Accident, Near Miss, Cat B, Cat C and Other (including Cat D and Other from above, mentioned 
for completeness but this categorization will not be a subject of modeling or expert elicitation). 
For the purposes of this modeling effort, the scope of attention is restricted to aircraft involved, 
Cat C and above RI events, with movement restrictions to be defined subsequently. For the 
purposes of this study, only two types of RI events are considered: 1) AC and AC and 2) AC and 
VEH. RI events include at least two objects [aircraft (AC), vehicle (VEH) and/or 
person/pedestrian (PED)] with one of the objects being the aircraft. 

As mentioned previously, the initial referenced RI database includes 10459 RI events (with no 
narratives). Among these, seven were accidents, 110 were near misses, 114 were CatB, 2014 were 
Cat C. Note that some RE events started as RI events and these were categorized as “not 
applicable” within the RI database; one such event occurred on August 27, 2006 where an aircraft 
crashed at the Lexington, KY airport resulting in 49 deaths 45 . The event was ultimately classified 
as an RE event, aircraft takeoff on wrong runway, but actually started as an RI event wherein the 
aircraft entered the takeoff runway (incorrect and too short) without authorization. This initial data 
set is useful for establishing overall probabilities related to the type of RI events that occur. The 
final data set consisted of 1596 RI events (CatC and above, with brief narratives). Of these, there 
was just one accident (the others among the 10459 were excluded because no narrative was 


13 



provided by the FAA), 30 were near misses, 20 were Cat B and 1545 were Cat C events. Of the 
1596 RI events, 1299 were caused by AC, 260 were caused by VEH and 37 were caused by PED. 
The complete data time frame ranges from 2001 through 2011, however, the narrative data time 
frame ranges from 2007 through 2011. The modeling time frame ranges from 2007 through 2014, 
or possibly 2015, at the latest. 

It is important to realize that RI events are “people intensive”, involving possibly two pilots 
and possibly two controllers (when some form of split control such as air / ground is in effect). A 
VEH driver or PED could replace one of the pilots. There are also organizations (FAA, Airport 
Management, private air transport companies, etc.) behind each of the people directly involved in 
the RI event. The Human Factors Analysis and Classification System (HFACS) by Wiegmann and 
Shappelll 46 are frequently used to describe the organizational, supervisory and personal factors 
states that establish preconditions for human errors and violations. Unfortunately, the narratives 
provided for the RI events do not provide sufficient detail for a standard HFACS assessment 
Hence, simplified HFACS models are used within this model, discussed subsequently. 

Furthermore, RI events are also “communication (Comm) intensive”; several instances of two- 
party communications (e.g., communications between pilot and ATC) must simultaneously 
function properly in order to avoid problems. Two-party communications involve both the content 
and transmission of information. The content must be correct and complete, timely and not too 
complex for the situation. The transmission must be accomplished without garbled or blocked 
information exchanges. Instances of two party communications exist between all the involved 
people in the RI event. Many times, communication frequencies are shared by numerous 
simultaneous two-party instances and confusion among all the parties can result when incorrect or 
incomplete information is transmitted correctly, or when correct information is not transmitted 
correctly. Taking this to the next level, split controllers are expected (by cockpit crews) to act as 
a unified controller and cockpit crews involving a pilot and co-pilot are expected (by control) to 
operate as a unified AC operator. Hence, any spilt entity needs adequate internal two party 
communications and adequate external two party communications must exist between the various 
entities. Failure of any part of this complex communication network results in deficient two party 
communications that can lead to confusion, a shared attribute among some or all of the participants . 
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The RI event severity rating is based strictly on the time / distance. The severity rating does not 
consider the object FAR vehicle classes, the potential for loss of life or damage to property, the 
causal path or error types, nor does it consider the HFACS states of the participants. In short, the 
existing FAA RI event definitions provide a very narrow way to examine runway safety, especially 
if the ultimate goal is to study the impact of technology injections. Many interesting questions can 
be posed and answered in the context of runway safety events that do not directly support RI event 
modeling by the strict FAA definitions; answers to these additional questions would provide 
significant insight into various aspects of possible technology injections and their effectiveness. 
During this work numerous alternative models have been developed, discussed and discarded 
either because they did not provide a satisfactory causal path, or because they were deemed to be 
too complex for use within the SME elicitation process. 

As noted previously, an attempt has been made to restrict the RI event scenarios of interest 
within this study. Part of this reduced scope involves movement restrictions for the objects 
involved. Object 1 (AC, VEH or PED) initiates the RI event and must be on the runway (RW) at 
the start of the RI event, or as an AC on final approach to the runway, having crossed the runway 
threshold. As an AC, Object 1 arrived (or will shortly arrive) on the RW either by incorrectly 
landing on it, taxiing onto it, or (in the case of crossing runways) the AC may be landing or taking 
off on one RW, while a second AC (Object 2) is using the second RW. If Object 1 is VEH or PED, 
it is assumed to be unauthorized agent of airport (an airport affiliated contractor or employee) that 
has moved onto the wrong runway or onto the correct runway but at the wrong time. Cases in 
which the VEH / PED arrived on the RW by uncontrolled, inappropriate runway access either 
directly via the airport perimeter, or indirectly through the airport terminal have been excluded 
from consideration. 

An RI event perspective versus an aviation perspective has been adopted. This means that every 
situation considered herein is assumed to result in an aircraft involved, Cat C or above RI event. 
Only controlled US airports are considered. The RI event time frame is assumed in the range from 
seconds to minutes. The FAR aircraft types of interest are Part 121 (Commercial) and Part 135 
(Air Taxi). These are considered together due to presumed similar equipment levels; this 
assumption was validated by the SME panel. Another aircraft type of interest, and a major 
contributor to RI events, is Part 91 (General Aviation). Discussion with the SME panel revealed 
that virtually any type of aircraft may be operated as a Part 91 vehicle; thus, the Part 91 distinction 
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is not very useful in this context. Other categories of AC (e.g., military, maintenance taxi) are 
included in the original data set, but these are not explicitly of interest in this study. The study 
considers pilot(s), controllers ), and relevant objects (vehicles and pedestrians) on the ground. The 
study also indirectly considers various airport geometries, various weather and visibility 
conditions, and various operating conditions. These factors are considered to be fixed during an 
RI event, whereas the participant HFACS states and two party communications are considered to 
be active during an RI event. Likewise, airport signs and markings are considered to be fixed 
mitigations during the RI event, whereas go-arounds, aborted takeoffs or other evasive maneuvers 
are considered to be active mitigating actions performed by the object operators. 

At this point, a preliminary BBN RI event model has been developed. An SME panel consisting 
of four consultants was assembled to review the model structure and to populate its content. The 
SME panel met over two days for about 14 hours of discussion about the complex RI event 
problem. A preliminary baseline RI model was agreed upon, as shown in Figure 2 and model 
population with SME likelihood belief values was also accomplished, discussed in a later section 
of this report. The nodes in Figure 2 are color coded to indicate associations among the various 
nodes. Generally, the flow of specific contributing factors through causal paths is from left to right 
in the figure. Many items funnel together through the two nodes identified as Fixed and Active 
Contributing Factors. The node identified as “RI Event Initiation” can be thought of as the start of 
the active mitigation phase of the RI event, which also ties back to the contributing factors. 
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Figure 2. The Runway Incursion Bayesian Belief Network. 


The SME panel validated many of the proposed definitions and most of the proposed model 
structure. However, the SME panel also provided significant clarification of several essential 
definitions within the RI event model. Moreover, the SME panel suggested several structural 
changes to the model, especially as related to the best way to model the active mitigation phase of 
the RI events. The overall complexity of the proposed RI event model was reduced from 39 nodes 
and a combined conditional probability table (CPT) size of 1041 elements to one of 37 nodes and 
combined CPT size of 735 elements. The remainder of this section describes the current 
preliminary RI event model. The node name for each is presented along with some clarifying 
comments. Most of the nodes are binary, meaning they have only two possible states: yes or no; 
where more states are present in a node, this will be made clear from the explanation. The goal of 
the SME elicitation is to provide probabilities for each of the possible states; for example, for the 
node “Airport Layout”, the SME goal is to determine the probability that the Airport Layout is an 
issue or not in the RI event. 
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The reader should first observe the numbers in Figure 2, representing the three possible primary 
participants in an RI event: 1 is the pilot or pilots (orange node), 2 is the air traffic controller (ATC) 
or controllers (cyan node) and 3 is an airport or contractor vehicle driver (pink node). Each of these 
nodes has a number of other color-coded nodes with lin ks pointing into these three primary nodes. 
Likewise, each of these three primary nodes have links pointing into the black node (Primary Error 
State). Starting with the green nodes (middle top), and moving counter-clockwise, the node 
descriptions of the RI event model follow: 

Airport Issues 

Airport Layout. The airport layout is an issue 16 - 18 . This may include potentially confusing 
elements such as parallel runways (with spacing of less than 1000 feet), intersecting runways, and 
taxiways parallel to and near runways, numerous taxiways crossing runways instead of perimeter 
taxiways. 

Signs, Markings and Lighting. The signs, markings and/or fixed equipment (e.g., lights) at the 
airport are deficient. This problem may be exacerbated under severe weather conditions when 
signs, etc. may be obscured from view 20 . 

Airport Construction or RW/TW Closure. Airport construction or runway/taxiway closure is 
an issue. 

Contamination Control. Contamination control, generally related to rain, snow or ice, is an 
issue 18 - 47 . 

Airport Issues. One or more of the issues within this grouping are present. 

ATC HFACS Issues 

Next are the purple nodes (upper left) which represent some of the most (as determined by the 
SME panel) Human Factors Analysis and Classification System 46 issues. The reader will note that 
a similar purple HFACS group accompanies both the pilot and the vehicle driver, though in the 
latter case, the HFACS group only includes training. 

ATC Cert Training Issues. Certification training for the ATC involved is an issue. 

ATC OTJ Training Issues. On the job training for the ATC involved is an issue as a distraction 13 . 
ATC Mental or Physical State. The current physical or mental state of the ATC involved is an 
issue. 
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ATC HFACS Issues. One or more of the HFACS issues within this grouping is an issue for the 
ATC involved 46 . 

ATC Operational Issues 

The cyan nodes (middle left) are other contributing factors that may influence the performance of 
the ATC. 

Automation Interaction Issues (ATC). Automation interaction is an issue for the ATC 
involved 48 . 

Abnormal Air Traffic Volume or Complexity. The traffic volume or complexity at this airport, 
at this time is an issue. For example, if the average traffic volume is high, it may cause a 
significantly increased work load for controllers and/or pilots; if low, it may result in extended 
periods of inactivity for controllers 16 . 

Staffing or Procedures Issues. The staffing level and/or work load management not appropriate 
for the situation is an issue, or the use of ambiguous or non-standardized ATC procedures is an 
issue 16 . 

ATC Operational Issues. One or more of the issues within this grouping are present. 

Two Party Communications 

The next group (blue nodes) describe the state of the system-wide two-party communications. 
Communication Content Issues. The completeness, correctness timeliness or complexity of 
communicated information is an issue. This may include the lack of a required usage for a call 
sign. Information may not have been transmitted at the appropriate time, i.e., it was not delayed 17 . 
Comm Hardware Error. Comm transmission is an issue. This may occur when the Comm system 
fails to operate as expected and may include blocked (“stepped on” communications where one 
party cuts off the communications of another), partially blocked (garbled or inaudible Comm 
transmission), hardware limitations /malfunctions and/or faulty headset jacks or connections 17 . 
Two Party Communication Issues. Comm Content Issues or a Communications Hardware Error 
has resulted in a Two Party Communications Error and is an issue. 
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Pilot HFACS Issues 

Another group of HFACS contributing factors is present for the pilot. The grouping is the same as 
before, although the relevant probabilities of these factors being an issue in an RI event may be 
different than for the ATC. 

Pilot Cert Training Issues. Certification training for the pilot(s) involved is an issue. 

Pilot OTJ Training Issues. On the job training for the pilot(s) involved is an issue as a 
distraction 13 . 

Pilot Mental or Physical State. The current physical or mental state of the pilot(s) involved is an 
issue. 

Pilot HFACS Issues. One or more of the HFACS issues within this grouping is an issue for the 
pilot(s) involved 46 . 

Other Pilot Operational Issues 

Other contributing factors that may influence the performance of the pilot(s) involved are show in 
the orange nodes. 

Inappropriate Aircraft Operations: Pilot operations of the aircraft, outside of the flight 
operational manual guidelines, is an issue causing the RI event. 

Automation Interaction Issues (Pilot): Automation interaction is an issue for the pilot(s) 
involved 48 . 

Pilot Operational Issues: One or more of the issues within this grouping are present. 

Driver Operational Issues 

These issues include the two nodes below, (purple and pink nodes, top right) 

Driver Training: The training of airport vehicle drivers is an issue 13 . 

Driver Operational Issues: One or more of the HFACS issues within this grouping is an issue 
for the vehicle driver(s) involved 46 . 

The preceding discussion covers all the nodes on the periphery of the left hand side of Figure 
2. These are all the issues potentially present that enable the RI event to occur. The nodes and 
states on the far right hand side of the figure generically define a specific RI event, of which 
numerous types and combinations may occur. NASA would hope to be in a position to broadly 
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address many, if not all. of these specific RI event types with technology injections. Starting with 
the black node of Figure 2 (middle right), the node descriptions follow. 

Primary Error State: The primary error source is either Controller Error (typically, loss of 
oversight), Pilot Error (typically, failure to hold short of a runway without authorization), or Other 
(includes mechanical failure and Driver Error, i.e., a failure to hold short of a runway without 
authorization). The SME panel excluded from consideration in this model non-airport authorized 
vehicles and all pedestrians on the runway. 

Collision Scenarios. The SME panel identified the most common collision scenarios: crossing in 
front of an aircraft on departure, crossing in front of an aircraft on arrival, or intersection events 
(crossing active runways) and other (everything else that leads to an RI event) 13 . 

Reaction time. This node has two states defined by the SME panel, short (eight seconds or less) 
and long (nine seconds or more) 

Final RI Event Severity. The RI event severity as would be reported by the FAA, including the 
impact of Contributing Factors and Mitigating Actions is established here. The states enumerated 
by the SME panel are accident / near miss, or other. Although less severe RI event categories have 
been defined by the FAA, these were deemed out of scope for this BBN model because the SME 
panel could not provide sufficient discrimination among these less severe RI events. 

This concludes the presentation of the RI event model. Likewise, the SME panel vetted many 
of the proposed definitions and most of the proposed model structure of the RE event model, shown 
in Figure 3. The SME panel again provided significant clarification of several essential definitions 
within the RE event model. Moreover, the SME panel suggested several simplifying structural 
changes to the model. The remainder of this section describes the current preliminary RE event 
model. The node name for each is presented along with some clarifying comments. Again, most 
of the nodes are binary with only two possible states (present as in issue in RE Events or not); 
where more states are present in a node, this will be made clear from the explanation of the node 
given subsequently. 

6.2 Runway Excursion Model 

An overrun is an RE event in which the aircraft departs the end of a runway; a veer-off is an 
RE event in which an aircraft departs the side of a runway. As an RE event may occur on landing 
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or takeoff, four types of RE events are possible: landing overrun, landing veer-off, takeoff overrun 
and takeoff veer-off. Some sources also include within RE events an aircraft attempting a landing 
that touches down in the undershoot area of the designated landing runway within the aerodrome 
perimeter 49 . 
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Figure 3. The Runway Excursion Bayesian Belief Network. 


Airport Issues 

Beginning with green nodes in the upper right corner of Figure 3, and moving counter-clockwise, 
the nodes and states are below. 

Approach and Departure Constraints. The physical or regulatory constraints on approach or 
departure trajectories for the airport in question are an issue. 

Contamination Control. Contamination control (e.g.,rain, snow or ice) for the airport in question 
is an issue. 

Runway Length. The runway length is an issue. This may be due to prevailing wind conditions, 
runway maintenance, or an aircraft landing on a runway that is too short for safe operations. 
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Airport Issues: One or more of the issues within this grouping are present. 


ATC HFACS Issues 

As in the RI event network, the ATC HFACS Issues group (purple nodes) is repeated in the RE 
event network. The SME panel rated the relative importance of these contributing factors for the 
ATC as being of much less consequence for RE events than for RI events. Next, the cyan nodes 
are described: 

ATC Operational Issues 

Runway Assignment. The runway assignment provided by ATC is an issue. This may be due to 
prevailing wind conditions, runway maintenance, or unusual airport operations. 

Runway Collision Avoidance. An RI event (typically failure to hold short of an active runway) 
has precipitated an RE event. This was noted by the SME panel as being an extremely rare 
occurrence. 

Contribution to Unstabilized Approach. The ATC has provided instructions that contribute to 
an unstabilized approach. 

Lack of Current Weather Information. The ATC involved have provided non-current weather 
information that contributes to an RE event. 

ATC Operational Issues. One or more of the issues within this grouping are present. 

Pilot HFACS Issues 

As in the RI event network, the Pilot HFACS Issues is repeated in the RE event network. The SME 
panel rated the relative importance of these contributing factors for the pilot as being about equal 
for RE and RI events. 
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Pilot Operational Issues and Remainder Nodes 

Inappropriate Aircraft Operations. Pilot operations of the aircraft, outside of the flight 
operational manual guidelines, is an issue causing the RE event. 

Unstabilized Approach. The pilot(s) involved have failed to perform a stabilized approach. 

Pilot Operational Issues. One or more of the issues within this grouping are present. 

Aircraft Automation Issues (blue node). Automation interaction is an issue for the pilot(s) 
involved 49 . 

Pilot Error. A pilot error has initiated an RE event. 

Weather Issues (yellow node). Weather issues have contributed to, or caused, an RE event. 
Mechanical Failure (pink node). Mechanical failure has contributed to, or caused, an RE event. 

RE Event Initiated: This node simply states whether an RE event has been initiated or not. 


7.0 Model Population 

7.1 Runway Incursion Model 

The subject matter expert (SME) model elicitation for the node Airport Layout is summarized 
in Table 1. Four SMEs, identified as SME1, SME2, SME3 and SME4, were used for the 
probability elicitation. 

Table 1. SME Probabilities and Confidence Bounds for the Node Airport Layout. 
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The first column identifies the state of the node as being an issue for RI events or not. Columns 
two through five provide the SME probabilities for each state. In column six, the minimum (Min) 
of the four SME responses is computed. In column seven, the maximum (Max) of the four SME 
responses is computed. In column eight, the average (Avg) of the four SME responses is 
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computed. In column nine, the standard deviation (StDev) of the four SME responses is computed. 
Columns 10 through 15 provide enclosed confidence bounds of the SME responses defined as: 

• Column 10: 99% Lo = Avg - 2.58*StDev 

• Column 11: 95% Lo = Avg - 1.96*StDev 

• Column 12: 90% Lo = Avg - 1.64*StDev 

• Column 13: 90% Hi = Avg + 1.64*StDev 

• Column 14: 95% Hi = Avg + 1.96*StDev 

• Column 15: 99% Hi = Avg + 2.58*StDev 

The multiplicative factors above were derived for the confidence intervals were based upon the 
Microsoft Excel function NORMINV(Prob,0,l), where Prob = 0.950, 0.975 and 0.995 for the 90%, 
95% and 99% confidence bounds above, respectively. These provide some meaningful ranges of 
the SME inputs to desired levels of confidence that can be used in sensitivity analysis studies. 

The SME model elicitation for the node Signs, Markings and Equipment is summarized in 
Table 2. 

Table 2. SME Probabilities and Confidence Bounds for the Node 
Signs, Markings and Equipment. 
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0.15 

0.15 

0.2 

0.16 

0.02 

0.1 

0.11 

0.12 

0.2 

0.21 

0.22 

not 

0.85 

0.8 

0.85 

0.85 

0.85 

0.8 

0.84 

0.98 

0.9 

0.89 

0.88 

0.8 

0.79 

0.78 


The SME model elicitation for the node Airport Construction or runway (RW) or taxiway (TW) 
Closure is summarized in Table 3. 
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Table 3. SME Probabilities and Confidence Bounds for the Node 
Airport Construction or RW/TW Closure. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.10 

0.10 

0.10 

0.15 

0.10 

0.15 

0.11 

0.02 

0.05 

0.06 

0.07 

0.15 

0.16 

0.17 

not 

0.90 

0.90 

0.90 

0.85 

0.90 

0.85 

0.89 

0.98 

0.95 

0.94 

0.93 

0.85 

0.84 

0.83 


The SME model elicitation for the node Contamination Control is summarized in Table 4. In 
this case, the 99%, 95% and 90% Lo confidence bounds would be less than zero. However, the 
computed values have been constrained to be greater than or equal to zero and less than or equal 
to unity. 


Table 4. SME Probabilities and Confidence Bounds for the Node 
Contamination Control. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.00 

0.05 

0.01 

0.01 

0.00 

0.05 

0.02 

0.02 

0.00 

0.00 

0.00 

0.06 

0.07 

0.08 

not 

1.00 

0.95 

0.99 

0.99 

1.00 

0.95 

0.98 

0.98 

1.00 

1.00 

1.00 

0.94 

0.93 

0.92 


The probabilities for the node Airport Issues are conditioned upon the presence or absence of 
the contributing factors in the leaf nodes which link into this node. That is to say the conditional 
probability table (CPT)in the node Airport Issues is conditioned upon the probabilities in the nodes 
1) Airport Layout, 2) Signs, Marking and Equipment, 3) Airport Construction or RW/TW Closure 
and 4) Contamination Control. Strictly speaking, the Airport Issues is an “or” node: if anyone of 
the contributing factors is present, then the result should be that airport issues are present. Such a 
CPT for a binary node (two output states, issues are present or issues are not present, highlighted 
in yellow) with two binary input factors is shown in Table 5. The reader should notice that all the 
probabilities are set to either zero or unity (highlighted in green). 
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Table 5. Conditional Probability Table for a Two-Factor Binary “or” Node. 


Factor 1 

Present 

Present 

Not Present 

Not Present 

Factor 2 

Present 

Not Present 

Present 

Not Present 

Present 

1 

1 

1 

0 

Not Present 

0 

0 

0 

1 


However, “or” nodes pose some difficulties for sensitivity analysis; hence a variety of other 
methods to populate the CPT for nodes like Airport Issues have been explored. In the current 
instance, the average SME probability for each of the four leaf nodes as being an issue that input 
to Airport Issues were summed, and then the probabilities were renormalized so as to sum to unity. 
That is, 0.31, 0.16, 0.11 and 0.02, for each of Airport Layout, Signs, Marking and Equipment, 
Airport Construction or RW/TW Closure and Contamination Control, respectively were summed 
(0.60) and then the probabilities were renormalized (0.52, 0.27, 0.18 and 0.03, respectively), so as 
to sum to unity. These renormalized probabilities were then summed to reflect the presence of the 
contributing factors. The resulting CPT for the Airport Issues node is shown in Table 6. A “yes” 
state indicates the factor is an issue, whereas a “no” state indicates that the factor is not an issue. 

Table 6. Conditional Probability Table for the Node Airport Issues. 


Factor 


1 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

no 

no 

no 

no 

2 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

3 

yes 

yes 

no 

no 

yes 

yes 

no 

no 

yes 

yes 

no 

no 

yes 

yes 

no 

no 

4 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 


State 


yes 

1.00 

0.97 

0.82 

0.78 

0.73 

0.70 

0.55 

0.52 

0.48 

0.45 

0.30 

0.27 

0.22 

0.18 

0.03 

0.00 

no 

0.00 

0.03 

0.18 

0.22 

0.27 

0.30 

0.45 

0.48 

0.52 

0.55 

0.70 

0.73 

0.78 

0.82 

0.97 

1.00 
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Other techniques for populating a CPT such as shown in Table 6 include: direct SME elicitation 
to determine combinatorial probabilities of the individual contributing factors, rank ordering, and 
weighting of the individual contributing factors. Direct elicitation of the SME to determine 
combinatorial probabilities (CPT values) of the individual contributing factors for a node like 
Airport Issues proved to be very difficult and time consuming for the RI model shown in Figure 

2. If the node under consideration has more than two states (such as the Primary Error State node, 
black, in Figure 2 with three states), this process became exponentially more tedious. What is 
desired is to get some measure of the relative importance of the individual contributing factors 
when considered in combination. The simplest approach is just to assume equal weighting for the 
contributing factors. Thus, for anode like Airport Issues, with four input nodes, the only possible 
values for the CPT are 0.00, 0.25, 0.50, 0.75 or 1.00, depending on how many of the four 
contributing factors are present. The next simplest approach may be just to have the SME panel 
rank order the contributing factors. For example the SME panel might (on average) rank the four 
contributing factors in this order: 

1. Sign, Marking and Equipment (most influential) 

2. Layout 

3. Construction or RW/TW Closure 

4. Contamination Control (least influential) 

The sum of the rank orderings is 10 (1 + 2 + 3+ 4= 10). The probabilities can then be 
renormalized and allocated in reverse order (0.4, 0.3, 0.2 and 0.1, respectively), so that the most 
influential contributing factor (Sign, Marking and Equipment) gets the greatest relative probability 
contribution (0.4). These relative importance probabilities can then be summed as described 
previously. This would lead to CPT values in the Airport Issues node having all the possible 
combinations of these values, as shown in Table 7. 
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Table 7. Conditional Probability Table based on rank ordering of contributing factors. 


Factor 


1 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

no 

No 

no 

no 

2 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

yes 

yes 

yes 

yes 

no 

No 

no 

no 

3 

yes 

yes 

no 

no 

yes 

yes 

no 

no 

yes 

yes 

no 

no 

yes 

Yes 

no 

no 

4 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

No 

yes 

no 


State 


yes 

1.00 

0.90 

0.80 

0.70 

0.60 

0.50 

0.40 

0.30 

0.70 

0.60 

0.50 

0.40 

0.30 

0.20 

0.10 

0.00 

no 

0.00 

0.10 

0.20 

0.30 

0.40 

0.50 

0.60 

0.70 

0.30 

0.40 

0.50 

0.60 

0.70 

0.80 

0.90 

1.00 


Finally, the SME panel can be asked to provide specific weightings for each of the 
contributing factors. If these weightings sum to unity, then no renormalization is necessary. 

The SME model elicitation for the node Air Traffic Control (ATC) Certification (Cert) 
Training Issues is summarized in Table 8. 

Table 8. SME Probabilities and Confidence Bounds for the Node 
ATC Cert Training Issues. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.01 

0.02 

0.01 

0.01 

0.01 

0.02 

0.01 

0.01 

0.00 

0.00 

0.00 

0.02 

0.02 

0.02 

not 

0.99 

0.98 

0.99 

0.99 

0.99 

0.98 

0.99 

1.00 

1.00 

1.00 

1.00 

0.98 

0.98 

0.98 


The SME model elicitation for the node ATC On-The-Job Training (OJT) Issues is 
summarized in Table 9. 

Table 9. SME Probabilities and Confidence Bounds for the Node 

ATC OJT Issues. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.05 

0.05 

0.08 

0.08 

0.05 

0.08 

0.06 

0.02 

0.02 

0.03 

0.03 

0.09 

0.09 

0.10 

not 

0.95 

0.95 

0.92 

0.93 

0.95 

0.92 

0.94 

0.98 

0.98 

0.97 

0.97 

0.91 

0.91 

0.90 
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The SME model elicitation for the node ATC Mental or Physical State Issues is summarized 
in Table 10. 


Table 10. SME Probabilities and Confidence Bounds for the Node 
ATC Mental or Physical State Issues. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.05 

0.10 

0.02 

0.10 

0.02 

0.10 

0.07 

0.04 

0.00 

0.00 

0.01 

0.13 

0.15 

0.17 

not 

0.95 

0.90 

0.98 

0.90 

0.98 

0.90 

0.93 

0.96 

1.00 

1.00 

0.99 

0.87 

0.85 

0.83 


The same method, described above, used for the node Airport Issues was also used to 
construct the CPT for the node ATC Human Factors Analysis and Classification System 
(HFACS) Issues. That is, the average probabilities for the three input nodes were summed and 
renormalized; the resulting normalized probabilities were then summed again to reflect the 
presence of the three contributing factors, as shown in Table 11. 

Table 11. Conditional Probability Table for the Node ATC HFACS Issues. 


state 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

ojt 

yes 

yes 

no 

no 

yes 

yes 

no 

no 

cert 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

ATC 

HFACS 

Issues 

1.00 

0.93 

0.57 

0.50 

0.50 

0.43 

0.07 

0.00 

No 

ATC 

HFACS 

Issues 

0.00 

0.07 

0.43 

0.50 

0.50 

0.57 

0.93 

1.00 


The SME model elicitation for the node Communication (Comm) Content Issues is 
summarized in Table 12. 
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Table 12. SME Probabilities and Confidence Bounds for the Node 

Conun Content Issues. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.25 

0.35 

0.30 

0.40 

0.25 

0.40 

0.33 

0.06 

0.16 

0.20 

0.22 

0.44 

0.46 

0.50 

not 

0.75 

0.65 

0.70 

0.60 

0.75 

0.60 

0.67 

0.94 

0.84 

0.80 

0.78 

0.56 

0.54 

0.50 


The SME model elicitation for the node Comm Hardware Issues is summarized in Table 13. 

Table 13. SME Probabilities and Confidence Bounds for the Node 
Comm Hardware Issues. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.01 

0.01 

0.00 

0.01 

0.00 

0.01 

0.01 

0.00 

0.00 

0.00 

0.00 

0.02 

0.02 

0.02 

not 

0.99 

0.99 

1.00 

1.00 

1.00 

0.99 

0.99 

1.00 

1.00 

1.00 

1.00 

0.98 

0.98 

0.98 


The same method, described above, used for the node Airport Issues was also used to 
construct the CPT for the node Two-Party Comm Issues. That is, the average probabilities for 
the two input nodes were summed and renormalized; the resulting normalized probabilities were 
then summed again to reflect the presence of the two contributing factors, as shown in Table 14. 


Table 14. Conditional Probability Table for the Node Two-Party Comm Issues. 


content 

yes 

yes 

no 

no 

hardware 

yes 

no 

yes 

no 

Two-Party 

Comm 

Issues 

1.00 

0.97 

0.03 

0.00 

No 

Issues 

0.00 

0.03 

0.97 

1.00 
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The SME model elicitation for the node Pilot Cert Training Issues is summarized in Table 15. 

Table 15. SME Probabilities and Confidence Bounds for the Node 
Pilot Cert Training Issues. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.01 

0.02 

0.02 

0.01 

0.01 

0.02 

0.02 

0.01 

0.01 

0.01 

0.01 

0.03 

0.03 

0.03 

not 

0.99 

0.98 

0.98 

0.99 

0.99 

0.98 

0.98 

0.99 

0.99 

0.99 

0.99 

0.97 

0.97 

0.97 


The SME model elicitation for the node Pilot OJT Issues is summarized in Table 16. 

Table 16. SME Probabilities and Confidence Bounds for the Node 

Pilot OJT Issues. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.05 

0.01 

0.01 

0.03 

0.01 

0.05 

0.03 

0.02 

0.00 

0.00 

0.00 

0.06 

0.07 

0.08 

not 

0.95 

0.99 

0.99 

0.97 

0.99 

0.95 

0.97 

0.98 

1.00 

1.00 

1.00 

0.94 

0.93 

0.92 


The SME model elicitation for the node Pilot Mental or Physical State Issues is summarized 
in Table 17. 


Table 17. SME Probabilities and Confidence Bounds for the Node 
Pilot Mental or Physical State Issues. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.05 

0.05 

0.02 

0.02 

0.02 

0.05 

0.04 

0.02 

0.00 

0.01 

0.01 

0.07 

0.07 

0.08 

not 

0.95 

0.95 

0.98 

0.98 

0.98 

0.95 

0.96 

0.98 

1.00 

0.99 

0.99 

0.93 

0.93 

0.92 


The same method, described above, used for the node Airport Issues was also used to 
construct the CPT for the node Pilot HFACS Issues. That is, the average probabilities for the 
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three input nodes were summed and renormalized; the resulting normalized probabilities were 
then summed again to reflect the presence of the three contributing factors, as shown in Table 18. 


Table 18. Conditional Probability Table for the Node Pilot HFACS Issues. 


state 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

ojt 

yes 

yes 

no 

no 

yes 

yes 

no 

no 

cert 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

Pilot 

HFACS 

Issues 

1.00 

0.78 

0.67 

0.44 

0.56 

0.33 

0.22 

0.00 

No 

Pilot 

HFACS 

Issues 

0.00 

0.22 

0.33 

0.56 

0.44 

0.67 

0.78 

1.00 


The SME model elicitation for the node Automation Interaction Issues (Pilot) is summarized 
in Table 19. 


Table 19. SME Probabilities and Confidence Bounds for the Node 
Automation Interaction Issues (Pilot). 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.03 

0.01 

0.05 

0.01 

0.01 

0.05 

0.02 

0.02 

0.00 

0.00 

0.00 

0.05 

0.06 

0.07 

not 

0.97 

0.99 

0.95 

1.00 

1.00 

0.95 

0.98 

0.98 

1.00 

1.00 

1.00 

0.95 

0.94 

0.93 
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The SME model elicitation for the node Inappropriate Aircraft Operations is summarized in 
Table 20. 


Table 20. SME Probabilities and Confidence Bounds for the Node 
Inappropriate Aircraft Operations. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.10 

0.10 

0.01 

0.10 

0.01 

0.10 

0.08 

0.05 

0.00 

0.00 

0.01 

0.15 

0.17 

0.20 

not 

0.90 

0.90 

0.99 

0.90 

0.99 

0.90 

0.92 

0.96 

1.00 

1.00 

0.99 

0.85 

0.83 

0.80 


The SME model elicitation for the node Automation Interaction Issues (ATC) is summarized 
in Table 21. 


Table 21. SME Probabilities and Confidence Bounds for the Node 
Automation Interaction Issues (ATC). 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.00 

0.05 

0.05 

0.10 

0.00 

0.10 

0.05 

0.04 

0.00 

0.00 

0.00 

0.12 

0.13 

0.15 

not 

1.00 

0.95 

0.95 

0.90 

1.00 

0.90 

0.95 

0.96 

1.00 

1.00 

1.00 

0.88 

0.87 

0.85 


The SME model elicitation for the node Abnormal Air Traffic Volume or Complexity is 
summarized in Table 22. 

Table 22. SME Probabilities and Confidence Bounds for the Node 
Abnormal Air Traffic Volume or Complexity. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.20 

0.20 

0.05 

0.25 

0.05 

0.25 

0.18 

0.09 

0.00 

0.01 

0.04 

0.32 

0.35 

0.40 

not 

0.80 

0.80 

0.95 

0.75 

0.95 

0.75 

0.82 

0.91 

1.00 

0.99 

0.96 

0.68 

0.65 

0.60 


34 












































































































































The SME model elicitation for the node Staffing or Procedural Issues is summarized in 
Table 23. 


Table 23. SME Probabilities and Confidence Bounds for the Node 
Staffing or Procedural Issues. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.05 

0.15 

0.20 

0.25 

0.05 

0.25 

0.16 

0.09 

0.00 

0.00 

0.02 

0.30 

0.33 

0.38 

not 

0.95 

0.85 

0.80 

0.75 

0.95 

0.75 

0.84 

0.91 

1.00 

1.00 

0.98 

0.70 

0.67 

0.62 


For the node ATC Operational Issues, a rank ordering of the relative effects of the six input 
nodes (Airport Issues, ATC HFACS Issues, Automation Interaction Issues (ATC), Abnormal Air 
Traffic Volume or Complexity, Staffing or Procedural Issues and Two-Party Comm Issues) was 
used. The average SME ranking of the six inputs was as follows: 

1. ATC HFACS Issues (most influential) 

2. Abnormal Air Traffic Volume or Complexity 

3. Staffing or Procedural Issues and Two-Party Comm Issues (tied) 

4. Airport Issues 

5. Automation Interaction Issues (ATC) (least influential) 

This Excel ranking (6, 5, 3.5, 3.5, 2 and 1, respectively) summed to 21, providing renormalized 
values of 0.29, 0.24, 0.17, 0.17, 0.10 and 0.05, respectively, highlighted in green in Table 24. The 
resulting 64 column CPT is too large to be legibly shown horizontally in this report. Hence, a 
transposed version of the CPT is shown in Table 24. The cells shown in green are the renormalized 
relative probabilities for the six input nodes that were summed in the column labeled “ATC Ops 
Issues” when a “yes” appeared in a row under the six input factors. 
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Table 24. Transposed Conditional Probability Table for the Node ATC Operational Issues 


0.10 

0.17 

0.29 

0.17 

0.24 

0.05 

ATC Ops 
Issues 

No ATC 
Ops 
Issues 

airport 

comm 

hfacs 

staffing 

volume 

auto 

yes 

yes 

yes 

yes 

yes 

yes 

1.00 

0.00 

yes 

yes 

yes 

yes 

yes 

no 

0.95 

0.05 

yes 

yes 

yes 

yes 

no 

yes 

0.76 

0.24 

yes 

yes 

yes 

yes 

no 

no 

0.71 

0.29 

yes 

yes 

yes 

no 

yes 

yes 

0.83 

0.17 

yes 

yes 

yes 

no 

yes 

no 

0.79 

0.21 

yes 

yes 

yes 

no 

no 

yes 

0.60 

0.40 

yes 

yes 

yes 

no 

no 

no 

0.55 

0.45 

yes 

yes 

no 

yes 

yes 

yes 

0.71 

0.29 

yes 

yes 

no 

yes 

yes 

no 

0.67 

0.33 

yes 

yes 

no 

yes 

no 

yes 

0.48 

0.52 

yes 

yes 

no 

yes 

no 

no 

0.43 

0.57 

yes 

yes 

no 

no 

yes 

yes 

0.55 

0.45 

yes 

yes 

no 

no 

yes 

no 

0.50 

0.50 

yes 

yes 

no 

no 

no 

yes 

0.31 

0.69 

yes 

yes 

no 

no 

no 

no 

0.26 

0.74 

yes 

no 

yes 

yes 

yes 

yes 

0.83 

0.17 

yes 

no 

yes 

yes 

yes 

no 

0.79 

0.21 

yes 

no 

yes 

yes 

no 

yes 

0.60 

0.40 

yes 

no 

yes 

yes 

no 

no 

0.55 

0.45 

yes 

no 

yes 

no 

yes 

yes 

0.67 

0.33 

yes 

no 

yes 

no 

yes 

no 

0.62 

0.38 

yes 

no 

yes 

no 

no 

yes 

0.43 

0.57 

yes 

no 

yes 

no 

no 

no 

0.38 

0.62 

yes 

no 

no 

yes 

yes 

yes 

0.55 

0.45 

yes 

no 

no 

yes 

yes 

no 

0.50 

0.50 

yes 

no 

no 

yes 

no 

yes 

0.31 

0.69 

yes 

no 

no 

yes 

no 

no 

0.26 

0.74 

yes 

no 

no 

no 

yes 

yes 

0.38 

0.62 

yes 

no 

no 

no 

yes 

no 

0.33 

0.67 

yes 

no 

no 

no 

no 

yes 

0.14 

0.86 

yes 

no 

no 

no 

no 

no 

0.10 

0.90 

no 

yes 

yes 

yes 

yes 

yes 

0.90 

0.10 

no 

yes 

yes 

yes 

yes 

no 

0.86 

0.14 

no 

yes 

yes 

yes 

no 

yes 

0.67 

0.33 

no 

yes 

yes 

yes 

no 

no 

0.62 

0.38 
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no yes yes no yes yes 0.74 0.26 

no yes yes no yes no 0.69 0.31 

no yes yes no no yes 0.50 0.50 

no yes yes no no no 0.45 0.55 

no yes no yes yes yes 0.62 0.38 

no yes no yes yes no 0.57 0.43 

no yes no yes no yes 0.38 0.62 

no yes no yes no no 0.33 0.67 

no yes no no yes yes 0.45 0.55 

no yes no no yes no 0.40 0.60 

no yes no no no yes 0.21 0.79 

no yes no no no no 0.17 0.83 

no no yes yes yes yes 0.74 0.26 

no no yes yes yes no 0.69 0.31 

no no yes yes no yes 0.50 0.50 

no no yes yes no no 0.45 0.55 

no no yes no yes yes 0.57 0.43 

no no yes no yes no 0.52 0.48 

no no yes no no yes 0.33 0.67 

no no yes no no no 0.29 0.71 

no no no yes yes yes 0.45 0.55 

no no no yes yes no 0.40 0.60 

no no no yes no yes 0.21 0.79 

no no no yes no no 0.17 0.83 

no no no no yes yes 0.29 0.71 

no no no no yes no 0.24 0.76 

no no no no no yes 0.05 0.95 

no no no no no no 0.00 1.00 

For the node Pilot Operational Issues, a hybrid approach was used to populate the CPT. The 
average SME input for the Inappropriate Aircraft Operations (0.08) and Automation Interaction 
Issues (Pilot) (0.02) were used. The remaining three input nodes to Pilot Operational Issues (e.g., 
Airport Issues, Two-Party Comm Issues and Pilot HFACS Issues) were already conditional nodes 
and no ranking ordering of these was performed. Hence, the average CPT value (0.5) was used as 
the weighting value for each of these three inputs to the Pilot Operational Issues node. The 
resulting renormalization of the weighting values (0.5, 0.5, 0.5 0.08 and 0.02 renormalized to 0.31, 
0.31, 0.31, 0.05 and 0.01, highlighted in green in Table 25) gives a disproportionate weight to the 
five inputs. The resulting CPT is shown (transposed) in Table 25. In retrospect, eliciting a rank 
ordering, or even using equal weighting, of the five inputs would have been a preferred method. 
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Table 25. Transposed Conditional Probability Table for the Node Pilot Operational Is sues 


0.31 

0.31 

0.31 

0.05 

0.01 

Pilot Ops 
Issues 

No Pilot 
Ops 
Issues 

airport 

comm 

hfacs 

in ops 

auto 

yes 

yes 

yes 

yes 

yes 

1.00 

0.00 

yes 

yes 

yes 

yes 

no 

0.99 

0.01 

yes 

yes 

yes 

no 

yes 

0.95 

0.05 

yes 

yes 

yes 

no 

no 

0.94 

0.06 

yes 

yes 

no 

yes 

yes 

0.69 

0.31 

yes 

yes 

no 

yes 

no 

0.68 

0.33 

yes 

yes 

no 

no 

yes 

0.64 

0.36 

yes 

yes 

no 

no 

no 

0.63 

0.38 

yes 

no 

yes 

yes 

yes 

0.69 

0.31 

yes 

no 

yes 

yes 

no 

0.68 

0.33 

yes 

no 

yes 

no 

yes 

0.64 

0.36 

yes 

no 

yes 

no 

no 

0.63 

0.38 

yes 

no 

no 

yes 

yes 

0.38 

0.63 

yes 

no 

no 

yes 

no 

0.36 

0.64 

yes 

no 

no 

no 

yes 

0.33 

0.68 

yes 

no 

no 

no 

no 

0.31 

0.69 

no 

yes 

yes 

yes 

yes 

0.69 

0.31 

no 

yes 

yes 

yes 

no 

0.68 

0.33 

no 

yes 

yes 

no 

yes 

0.64 

0.36 

no 

yes 

yes 

no 

no 

0.63 

0.38 

no 

yes 

no 

yes 

yes 

0.38 

0.63 

no 

yes 

no 

yes 

no 

0.36 

0.64 

no 

yes 

no 

no 

yes 

0.33 

0.68 

no 

yes 

no 

no 

no 

0.31 

0.69 

no 

no 

yes 

yes 

yes 

0.38 

0.63 

no 

no 

yes 

yes 

no 

0.36 

0.64 

no 

no 

yes 

no 

yes 

0.33 

0.68 

no 

no 

yes 

no 

no 

0.31 

0.69 

no 

no 

no 

yes 

yes 

0.06 

0.94 

no 

no 

no 

yes 

no 

0.05 

0.95 

no 

no 

no 

no 

yes 

0.01 

0.99 

no 

no 

no 

no 

no 

0.00 

1.00 
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The SME model elicitation for the node Driver Training is summarized in Table 26. 


Table 26. SME Probabilities and Confidence Bounds for the Node 

Driver Training. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.03 

0.01 

0.04 

0.03 

0.01 

0.04 

0.03 

0.01 

0.00 

0.01 

0.01 

0.05 

0.05 

0.06 

not 

0.97 

0.99 

0.96 

0.97 

0.99 

0.96 

0.97 

0.99 

1.00 

0.99 

0.99 

0.95 

0.95 

0.94 


For the node Driver Operational Issues, the hybrid approach (described above) was again used 
to populate the CPT. The average SME input (0.03) for the node Driver Training was used along 
with the average CPT value (0.5) for the nodes Airport Issues and Two-Party Comm Issues. The 
resulting renormalization of the weighting values (0.5, 0.5, and 0.03 renormalized to 0.49, 0.49 
and 0.03) gives a disproportionate weight to the three inputs. The resulting CPT is shown in Table 
27. In retrospect, eliciting a rank ordering, or even using equal weighting, of the five inputs would 
have been a preferred method. 

Table 27. Conditional Probability Table for the Node Driver Operational Issues. 


training 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

comm 

yes 

yes 

no 

no 

yes 

yes 

no 

no 

airport 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

driver 

ops 

issues 

1.00 

0.51 

0.51 

0.03 

0.97 

0.49 

0.49 

0.00 

no driver 
ops 
issues 

0.00 

0.49 

0.49 

0.97 

0.03 

0.51 

0.51 

1.00 


The node Primary Error State has three input links (Driver Operational Issues, ATC Operational 
Issues and Pilot Operational Issues) and three output states: Pilot Error, Controller Error and Other. 
Unfortunately, the SME panel provided probabilities for a different version of this table in which 
four output states were possible: Pilot Error, Controller Error, Driver Error and Other (mechanical 
failure); subsequent discussions removed the “Other” state from this node and removed from 
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consideration any RI caused by mechanical failure. However, since the elicitation results did not 
align with the final version of table, a decision was made to combine the elicitation results for the 
Driver Error state with those for the Other state, and to rename the third output state as Other 
(includes Driver Error and mechanical failure). The resulting CPT has 3 rows and 8 columns. 
Each column describes the probabilities for a combination of Driver Operational Issues, Pilot 
Operational Issues and ATC Operational Issues; these combinations are highlighted in orange, 
yellow and green as shown in Table 28. 

Table 28. SME Probabilities and Confidence Bounds for the Node 

Primary Error State. 



driver issues 

yes 

pilot issues 

yes 

controller issues 

yes 

Min 

Max 

Avg 

99% Lo 

95% Lo 

90% Lo 

90% Hi 

95% Hi 

99% Hi 

pilot error 

0.30 

0.49 

0.44 

0.19 

0.25 

0.28 

0.60 

0.63 

0.69 

controller error 

0.20 

0.20 

0.20 

0.20 

0.20 

0.20 

0.20 

0.20 

0.20 

other 

0.31 

0.50 

0.36 

0.11 

0.17 

0.20 

0.52 

0.55 

0.61 


driver issues 

yes 

pilot issues 

yes 

controller issues 

no 

Min 

Max 

Avg 

99% Lo 

95% Lo 

90% Lo 

90% Hi 

95% Hi 

99% Hi 

pilot error 

0.45 

0.70 

0.59 

0.32 

0.39 

0.42 

0.76 

0.79 

0.86 

controller error 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

other 

0.30 

0.55 

0.42 

0.13 

0.20 

0.23 

0.61 

0.64 

0.71 


driver issues 

yes 

pilot issues 

no 

controller issues 

yes 

Min 

Max 

Avg 

99% Lo 

95% Lo 

90% Lo 

90% Hi 

95% Hi 

99% Hi 

pilot error 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

controller error 

0.35 

0.40 

0.38 

0.31 

0.32 

0.33 

0.43 

0.44 

0.45 

other 

0.59 

0.66 

0.63 

0.54 

0.56 

0.57 

0.69 

0.70 

0.72 


driver issues 

yes 

pilot issues 

no 

controller issues 

no 

Min 

Max 

Avg 

99% Lo 

95% Lo 

90% Lo 

90% Hi 

95% Hi 

99% Hi 

pilot error 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

controller error 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

other 

0.99 

1.00 

1.00 

0.99 

0.99 

0.99 

1.00 

1.00 

1.00 


driver issues 

no 

pilot issues 

yes 

controller issues 

yes 

Min 

Max 

Avg 

99% Lo 

95% Lo 

90% Lo 

90% Hi 

95% Hi 

99% Hi 

pilot error 

0.60 

0.70 

0.65 

0.50 

0.54 

0.56 

0.74 

0.76 

0.80 

controller error 

0.25 

0.39 

0.32 

0.16 

0.20 

0.22 

0.42 

0.44 

0.48 

other 

0.01 

0.05 

0.03 

0.00 

0.00 

0.00 

0.07 

0.08 

0.09 


driver issues 

no 

pilot issues 

yes 

controller issues 

no 

Min 

Max 

Avg 

99% Lo 

95% Lo 

90% Lo 

90% Hi 

95% Hi 

99% Hi 

pilot error 

0.99 

1.00 

0.99 

0.98 

0.98 

0.98 

1.00 

1.00 

1.00 

controller error 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 
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other 

0.00 

0.01 

0.01 

0.00 

0.00 

0.00 

0.02 

0.02 

0.02 


driver issues 

no 

pilot 

ssues 

no 

controller issues 

yes 


Min 

Max 

Avg 

99% Lo 

95% Lo 

90% Lo 

90% Hi 

95% Hi 

99% Hi 

pilot error 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

controller error 

0.99 

1.00 

0.99 

0.98 

0.98 

0.98 

1.00 

1.00 

1.00 

other 

0.00 

0.01 

0.01 

0.00 

0.00 

0.00 

0.02 

0.02 

0.02 


driver issues 

no 

pilot 

ssues 

no 

controller issues 

no 


Min 

Max 

Avg 

99% Lo 

95% Lo 

90% Lo 

90% Hi 

95% Hi 

99% Hi 

pilot error 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

controller error 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

0.00 

other 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 


The SME model elicitation for the node Collision Scenarios is summarized in Table 29. 

Table 29. SME Probabilities and Confidence Bounds for the Node 

Collision Scenarios. 


Collision 

Scenarios 

SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

crossing / 
departure 

0.10 

0.15 

0.20 

0.15 

0.10 

0.20 

0.15 

0.04 

0.04 

0.07 

0.08 

0.22 

0.23 

0.26 

crossing / 
arrival 

0.10 

0.15 

0.10 

0.20 

0.10 

0.20 

0.14 

0.05 

0.02 

0.05 

0.06 

0.22 

0.23 

0.26 

intersecting 

runways 

0.05 

0.05 

0.05 

0.10 

0.05 

0.10 

0.06 

0.03 

0.00 

0.01 

0.02 

0.10 

0.11 

0.12 

other 

0.75 

0.65 

0.65 

0.55 

0.55 

0.75 

0.65 

0.08 

0.44 

0.49 

0.52 

0.78 

0.81 

0.86 


The SME model elicitation for the node Reaction Time is summarized in Table 30. 

Table 30. SME Probabilities and Confidence Bounds for the Node 

Reaction Time. 


reaction 

time 

SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

short 
(8 sec or 
less) 

0.30 

0.40 

0.03 

0.05 

0.03 

0.40 

0.20 

0.18 

0.00 

0.00 

0.00 

0.50 

0.56 

0.67 
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The node Potential RI Event Severity is conditioned upon the nodes Primary Error State, 
Collision Scenario and Reaction Time. A direct CPT elicitation was used and the average 
probabilities for the node are presented in Table 31. In Table 31, the following abbreviations are 
used: “cont” refers to controller error and“xrw” means crossing runways. 


Table 31. Transposed Conditional Probability Table for the Node Potential RI Event 

Severity. 








































































































































7.2 Runway Excursion Model 


The SME model elicitation for the node Approach and Departure Constraints is summarized in 
Table 32. 


Table 32. SME Probabilities and Confidence Bounds for the Node 
Approach and Departure Constraints. 



SME1 

SME2 

SME2 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.05 

0.10 

0.05 

0.03 

0.03 

0.10 

0.06 

0.03 

0.00 

0.00 

0.00 

0.11 

0.12 

0.14 

not 

0.95 

0.90 

0.95 

0.98 

0.98 

0.90 

0.94 

0.97 

1.00 

1.00 

1.00 

0.89 

0.88 

0.86 


The SME model elicitation for the node Contamination Control is summarized in Table 33. 

Table 33. SME Probabilities and Confidence Bounds for the Node 

Conta mi nation Control. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.05 

0.05 

0.00 

0.01 

0.00 

0.05 

0.03 

0.03 

0.00 

0.00 

0.00 

0.07 

0.08 

0.09 

not 

0.95 

0.95 

1.00 

0.99 

1.00 

0.95 

0.97 

0.97 

1.00 

1.00 

1.00 

0.93 

0.92 

0.91 


The SME model elicitation for the node Runway Length is summarized in Table 34. 

Table 34. SME Probabilities and Confidence Bounds for the Node 

Runway Length. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.10 

0.15 

0.10 

0.15 

0.10 

0.15 

0.13 

0.03 

0.05 

0.07 

0.08 

0.17 

0.18 

0.20 

not 

0.90 

0.85 

0.90 

0.85 

0.90 

0.85 

0.87 

0.97 

0.95 

0.93 

0.92 

0.83 

0.82 

0.80 
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The average SME input for node Airport Issues is shown in Table 34. The following 
abbreviations are used in Table 35: app =approach, dep = departure, cont = contamination 
control and rw = runway. 


Table 35. Conditional Probability Table for the node Airport Issues. 


app/dep 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

cont 

yes 

yes 

no 

no 

yes 

yes 

no 

no 

rw 

length 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

issue 

0.99 

0.61 

0.71 

0.28 

0.82 

0.19 

0.59 

0.01 

not 

0.01 

0.39 

0.29 

0.72 

0.18 

0.81 

0.41 

0.99 


The SME model elicitation for the node Runway Assignment is summarized in Table 36. 

Table 36. SME Probabilities and Confidence Bounds for the Node 

Runway Assignment. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.05 

0.05 

0.01 

0.05 

0.01 

0.05 

0.04 

0.02 

0.00 

0.00 

0.01 

0.07 

0.08 

0.09 

not 

0.95 

0.95 

0.99 

0.95 

0.99 

0.95 

0.96 

0.98 

1.00 

1.00 

0.99 

0.93 

0.92 

0.91 


The SME model elicitation for the node Runway Collision Avoidance is summarized in 
Table 37. 


Table 37. SME Probabilities and Confidence Bounds for the Node 
Runway Collision Avoidance. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.00 

0.01 

0.01 

0.01 

0.00 

0.01 

0.01 

0.00 

0.00 

0.00 

0.00 

0.01 

0.02 

0.02 

not 

1.00 

0.99 

1.00 

0.99 

1.00 

0.99 

0.99 

1.00 

1.00 

1.00 

1.00 

0.99 

0.98 

0.98 


44 













































































































































The SME model elicitation for the node Contribution to Unstabilized Approach is summarized 
in Table 38. 


Table 38. SME Probabilities and Confidence Bounds for the Node 
Contribution to Unstabilized Approach. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.10 

0.10 

0.20 

0.25 

0.10 

0.25 

0.16 

0.08 

0.00 

0.02 

0.04 

0.29 

0.31 

0.36 

not 

0.90 

0.90 

0.80 

0.75 

0.90 

0.75 

0.84 

0.93 

1.00 

0.98 

0.96 

0.71 

0.69 

0.64 


The SME model elicitation for the node Lack of Current Weather Information is summarized 
in Table 39. 


Table 39. SME Probabilities and Confidence Bounds for the Node 
Lack of Current Weather Info. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.10 

0.01 

0.01 

0.10 

0.01 

0.10 

0.06 

0.05 

0.00 

0.00 

0.00 

0.14 

0.16 

0.19 

not 

0.90 

0.99 

0.99 

0.90 

0.99 

0.90 

0.94 

0.95 

1.00 

1.00 

1.00 

0.86 

0.84 

0.81 


The average SME model elicitation for the node ATC Operational Issues is summarized in 
Table 40. The HFACS contribution to the CPT are considered separately. The following 
abbreviations are used in Table 40: hfacs = human factors assessment and classification system 
rw = runway, cntrb = contribution, unstab = unstabilized, app = approach, and curr = current. 

Table 40. Transposed Conditional Probability Table for the node 
ATC Operational Issues. 


hfacs 

rw assign 

coll avoid 

cntrb to 
unstab app 

lack of curr 
weather 

ATC Ops 
Issues 

No ATC 
Ops Issues 

yes 

yes 

yes 

yes 

yes 

1.0000 

0.0000 

yes 

yes 

yes 

yes 

no 

0.8705 

0.1295 

yes 

yes 

yes 

no 

yes 

0.5702 

0.4298 


45 



























































































































yes 

no 


no 


no 


yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

no 

no 

no 

no 

no 

no 

no 

no 

no 

no 

no 

no 


yes 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

no 

no 

no 

no 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

no 

no 

no 

no 


no 

no 

no 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

yes 

yes 

yes 

yes 

no 

no 

no 

no 

yes 

yes 

yes 

yes 

no 

no 

no 

no 


yes 

yes 

no 

no 

yes 

yes 

no 

no 

yes 

yes 

no 

no 

yes 

yes 

no 

no 

yes 

yes 

no 

no 

yes 

yes 

no 

no 

yes 

yes 

no 

no 


yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 

yes 

no 


0.4311 0.5689 

0.9039 0.0961 

0.7370 0.2630 

0.5006 0.4994 

0.1530 0.8470 

0.8761 0.1239 

0.6536 0.3464 

0.4172 0.5828 

0.2086 0.7914 

0.8066 0.1934 

0.6258 0.3742 

0.1418 0.8582 

0.0147 0.9853 

0.9925 0.0075 

0.7825 0.2175 

0.5125 0.4875 

0.3875 0.6125 

0.8125 0.1875 

0.6625 0.3375 

0.4500 0.5500 

0.1375 0.8625 

0.7875 0.2125 

0.5875 0.4125 

0.3750 0.6250 

0.1875 0.8125 

0.7250 0.2750 

0.5625 0.4375 

0.1275 0.8725 

0.0133 0.9868 


The SME model elicitation for the node Unstabilized Approach is summarized in Table 41. 


Table 41. SME Probabilities and Confidence Bounds for the Node 

Unstabilized Approach. 


with automation issue 


SMEl 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.30 

0.25 

0.25 

0.50 

0.25 

0.50 

0.36 

0.13 

0.03 

0.11 

0.15 

0.57 

0.61 

0.69 

not 

0.70 

0.75 

0.75 

0.50 

0.75 

0.50 

0.64 

0.87 

0.97 

0.89 

0.85 

0.43 

0.39 

0.31 
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with no automation issue 


SMEl 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.01 

0.01 

0.10 

0.15 

0.01 

0.15 

0.07 

0.07 

0.00 

0.00 

0.00 

0.18 

0.20 

0.25 

not 

0.99 

0.99 

0.90 

0.85 

0.99 

0.85 

0.93 

0.93 

1.00 

1.00 

1.00 

0.82 

0.80 

0.75 


The SME model elicitation for the node Aircraft Automation Issues is summarized in Table 42. 

Table 42. SME Probabilities and Confidence Bounds for the Node 
Aircraft Automation Issues. 



SMEl 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.01 

0.01 

0.10 

0.15 

0.01 

0.15 

0.07 

0.07 

0.00 

0.00 

0.00 

0.18 

0.20 

0.25 

not 

0.99 

0.99 

0.90 

0.85 

0.99 

0.85 

0.93 

0.93 

1.00 

1.00 

1.00 

0.82 

0.80 

0.75 


The SME model elicitation for the node Inappropriate Aircraft Operations is summarized in 
Table 43. 


Table 43. SME Probabilities and Confidence Bounds for the Node 
Inappropriate Aircraft Operations. 



SMEl 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

issue 

0.30 

0.10 

0.10 

0.30 

0.10 

0.30 

0.20 

0.12 

0.00 

0.00 

0.01 

0.39 

0.43 

0.50 

not 

0.70 

0.90 

0.90 

0.70 

0.90 

0.70 

0.80 

0.88 

1.00 

1.00 

0.99 

0.61 

0.57 

0.50 


The average SME model elicitation for the node Pilot Operational Issues is summarized in 
Table 44. The HFACS contribution to the CPT is considered separately. 
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Table 44. Transposed Conditional Probability Table for the node 
Pilot Operational Issues. 


unstab 

app 

pilot 

hfacs 

inapp ac 
ops 

auto 

Pi lot Ops 
Issues 

No Pilot 
Ops 
Issues 

yes 

yes 

yes 

yes 

0.99 

0.01 

yes 

yes 

yes 

no 

0.83 

0.18 

yes 

yes 

no 

yes 

0.79 

0.21 

yes 

yes 

no 

no 

0.63 

0.38 

yes 

no 

yes 

yes 

0.81 

0.19 

yes 

no 

yes 

no 

0.74 

0.26 

yes 

no 

no 

yes 

0.50 

0.50 

yes 

no 

no 

no 

0.39 

0.61 

no 

yes 

yes 

yes 

0.59 

0.41 

no 

yes 

yes 

no 

0.45 

0.55 

no 

yes 

no 

yes 

0.46 

0.54 

no 

yes 

no 

no 

0.15 

0.85 

no 

no 

yes 

yes 

0.55 

0.45 

no 

no 

yes 

no 

0.39 

0.61 

no 

no 

no 

yes 

0.23 

0.78 

no 

no 

no 

no 

0.01 

0.99 


The SME panel developed blanket multiplication factors (mfac) for the HFACS contributors 
for RE events relative to those previously recorded for RI events. The SME elicitation for the 
ATC and pilot multiplicative factors is summarized in Table 45. 
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Table 45. SME multiplicative factors and confidence bounds for HFACS contributions 

to RE relative to RI events. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

ate 

hfacs 

mfac 

0.10 

0.20 

0.10 

0.20 

0.10 

0.20 

0.15 

0.06 

0.00 

0.04 

0.06 

0.24 

0.26 

0.30 

pilot 

hfacs 

mfac 

0.80 

1.10 

1.0 

1.0 

0.80 

1.10 

0.98 

0.13 

0.65 

0.73 

0.77 

1.00 

1.00 

1.00 


The revised marginal probabilities for the RE / ATC / HFACS contributions are summarized 
in Table 46. 


Table 46. RE / ATC / HFACS contributions. 


RE 

HFACS 

issue 

not 

Avg 

Mfac 

issue 

not 

ATC Cert 

0.0100 

0.9900 

0.1500 

0.0015 

0.9985 

ATC OTJ 

0.0600 

0.9400 

0.1500 

0.0090 

0.9910 

ATC 

State 

0.0700 

0.9300 

0.1500 

0.0105 

0.9895 


The revised marginal probabilities for the RE / Pilot / HFACS contributions are summarized 
in Table 47. 


Table 47. RE / Pilot / HFACS contributions. 


RE 

HFACS 

issue 

not 

Avg 

Mfac 

issue 

not 

Pilot 

Cert 

0.0200 

0.9800 

0.9800 

0.0196 

0.9804 

Pilot OTJ 

0.0300 

0.9700 

0.9800 

0.0294 

0.9706 

Pilot 

State 

0.0400 

0.9600 

0.9800 

0.0392 

0.9608 
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The average SME model elicitation for the node Pilot Error is summarized in Table 48. 


Table 48. Transposed Conditional Probability Table for the node 

Pilot Error. 


airport 

weather 

ate 

pilot 

Pilot 

No Pilot 

issues 

issues 

issues 

issues 

Error 

Error 

yes 

yes 

yes 

yes 

0.80 

0.20 

yes 

yes 

yes 

no 

0.31 

0.69 

yes 

yes 

no 

yes 

0.65 

0.35 

yes 

yes 

no 

no 

0.19 

0.81 

yes 

no 

yes 

yes 

0.60 

0.40 

yes 

no 

yes 

no 

0.16 

0.84 

yes 

no 

no 

yes 

0.38 

0.62 

yes 

no 

no 

no 

0.08 

0.92 

no 

yes 

yes 

yes 

0.71 

0.29 

no 

yes 

yes 

no 

0.16 

0.84 

no 

yes 

no 

yes 

0.63 

0.37 

no 

yes 

no 

no 

0.07 

0.93 

no 

no 

yes 

yes 

0.48 

0.52 

no 

no 

yes 

no 

0.03 

0.97 

no 

no 

no 

yes 

0.32 

0.68 

no 

no 

no 

no 

0.00 

1.00 
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The SME model elicitation for the node Mechanical Failure is summarized in Table 49. 


Table 49. SME Probabilities and Confidence Bounds for the Node 

Mechanical Failure. 



automation 

yes 


SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

mech 

failure 

0.2 

0.25 

N/A 

0.30 

0.20 

0.30 

0.25 

0.05 

0.12 

0.15 

0.17 

0.33 

0.35 

0.38 

no 

failure 

0.8 

0.75 

N/A 

0.70 

0.80 

0.70 

0.75 

0.95 

0.88 

0.85 

0.83 

0.67 

0.65 

0.62 


automation 

no 


SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

mech 

failure 

0.05 

0.05 

0.05 

0.05 

0.05 

0.05 

0.05 

0.00 

0.05 

0.05 

0.05 

0.05 

0.05 

0.05 

no 

failure 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 

1.00 

0.95 

0.95 

0.95 

0.95 

0.95 

0.95 
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The SME model elicitation for the node RE Event Initiated is summarized in Table 50. 


Table 50. SME Probabilities and Confidence Bounds for the Node 

RE Event Initiated. 



Mechanical Failure 

yes 


Pilot Error 

yes 











99% 

95% 

90% 

90% 

95% 

99% 


SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

Lo 

Lo 

Lo 

Hi 

Hi 

Hi 

RE Event 

0.90 

0.99 

0.99 

0.95 

0.90 

0.99 

0.96 

0.04 

0.85 

0.87 

0.89 

1.00 

1.00 

1.00 

No 

0.10 

0.01 

0.01 

0.05 

0.10 

0.01 

0.04 

0.96 

0.15 

0.13 

0.11 

0.00 

0.00 

0.00 

Event 
















Mechanical Failure 

yes 


Pilot Error 

no 











99% 

95% 

90% 

90% 

95% 

99% 


SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

Lo 

Lo 

Lo 

Hi 

Hi 

Hi 

RE Event 

0.70 

0.55 

0.75 

0.70 

0.55 

0.75 

0.68 

0.09 

0.45 

0.51 

0.53 

0.82 

0.84 

0.90 

No 

0.30 

0.45 

0.25 

0.30 

0.45 

0.25 

0.33 

0.91 

0.55 

0.49 

0.47 

0.18 

0.16 

0.10 

Event 
















Mechanical Failure 

no 


Pilot Error 

yes 











99% 

95% 

90% 

90% 

95% 

99% 


SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

Lo 

Lo 

Lo 

Hi 

Hi 

Hi 

RE Event 

0.60 

0.40 

0.40 

0.55 

0.40 

0.60 

0.49 

0.10 

0.22 

0.29 

0.32 

0.66 

0.69 

0.75 

No 

0.40 

0.60 

0.60 

0.45 

0.60 

0.40 

0.51 

0.90 

0.78 

0.71 

0.68 

0.34 

0.31 

0.25 

Event 
















Mechanical Failure 

no 


Pilot Error 

no 











99% 

95% 

90% 

90% 

95% 

99% 


SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

Lo 

Lo 

Lo 

Hi 

Hi 

Hi 

RE Event 

0.01 

0.01 

0.01 

0.01 

0.01 

0.01 

0.01 

0.00 

0.01 

0.01 

0.01 

0.01 

0.01 

0.01 

No 

0.99 

0.99 

0.99 

0.99 

0.99 

0.99 

0.99 

1.00 

0.99 

0.99 

0.99 

0.99 

0.99 

0.99 

Event 
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The SME model elicitation for the node Weather Issues is summarized in Table 51. 


Table 51. SME Probabilities and Confidence Bounds for the Node 

Weather Issues. 



SME1 

SME2 

SME3 

SME4 

Min 

Max 

Avg 

St Dev 

99% 

Lo 

95% 

Lo 

90% 

Lo 

90% 

Hi 

95% 

Hi 

99% 

Hi 

Issue 

0.85 

0.7 

0.75 

0.7 

0.70 

0.85 

0.75 

0.07 

0.57 

0.61 

0.63 

0.87 

0.89 

0.93 

Not 

0.15 

0.3 

0.25 

0.3 

0.30 

0.15 

0.25 

0.93 

0.43 

0.39 

0.37 

0.13 

0.11 

0.07 


8.0 Model Baseline Execution 

8.1 Runway Incursion Model 

The most recent SME session (July 2014) resulted in a fully vetted BBN model for both RI 
and RE events. Moreover, the SME panel elicitation of marginal and conditional probabilities has 
also been completed. Thus, all the ingredients for a fully vetted and fully populated set of baseline 
models have been obtained. In order to demonstrate how a BBN model would function, random 
values for all of the conditional probability tables have been inserted into the final RI model so 
that model operations can be simulated as shown in Figure 4. The node probabilities are shown 
overlaying the model structure from Figure 2. In this case, the marginal (leaf node) probabilities 
for all of the contributing factors have been set to zero, and the reaction time has been set to “long” 
(i.e., more than 8 seconds) for the collision scenario “crossing in front of departure”. The model 
indicates that the probability of an accident or near miss (red node at middle right) is about 22% 
for this random scenario. 
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Figure 4. Sample RI Model Execution (baseline scenario). 
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Figure 5. Sample RI Model Execution (worst-case scenario). 


In Figure 5, the same example has been shown again, but with all the leaf node marginal 
probabilities set to 1 (all the issue are a certainty) and the reaction time has been set to short for 
the same collision scenario. In this random, worst-case, the probability of an accident or near miss 
is increased to about 75%. 


8.2 Runway Excursion Model 

Figure 6 illustrates a sample execution of the RE model with average probabilities employed, 
as described in section 6 of this report. The node probabilities are again shown as an overlay to 
the model structure presented in Figure 3. The sample execution shown in Figure 6 indicates that 
under the assumptions used to construct this model, there is about a 10% chance of an RE event. 
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Figure 6. Sample RE Model Execution (average probabilities). 


9.0 Model Sensitivity Analysis 

9.1 Runway Incursion Model 

Figure 7 illustrates a typical sensitivity analysis for the RI model. In order to accomplish this 
sensitivity analysis, the Hugin tool parametrically varies the strength of the marginal and 
conditional probability table values. Then sensitivity values are determined at the maximum 
strength, the minimum strength and at the baseline (or average) CPT values. In figure 7 below 
those nodes where blue coloring is observed indicate response sensitivity at the maximum CPT 
values, those with red coloring indicate response sensitivity at the minimum CPT values, and 
those with green coloring indicate response sensitivity at the average (or baseline) CPT values. 

In each case, the intensity of the observed colors indicate the strength of the observed response 
sensitivity. Under the assumptions present in the model (i.e., nodes, links and CPT values as 
previously discussed), the node Potential RI Event Severity is, as expected, sensitive to the node 
Primary Error State; however, only very limited sensitivity discrimination among the other nodes 
to the left is possible. Yet, some degree of sensitivity must exist for every node in the diagram. 


56 


Thus, in order to examine more fully the sensitivities across the entire diagram, Figure 7 
illustrates only the sensitivities for the node Primary Error State. The strongest sensitivities for 
this node are with respect to the Pilot Operational Issues, the Pilot Physical or Mental State, 
Comm Content Issues, and the Airport Layout. 



Figure 7. Sample RI model execution sensitivity analysis. 


9.2 Runway Excursion Model 

Figure 8 illustrates a typical sensitivity analysis for the RE model. Under the assumptions 
present in the model (nodes, lin ks and CPT as previously discussed), Figure 8 illustrates that an 
RE Event Initiated is mostly dependent upon mechanical failure and then upon pilot error, 
runway length and automation issues. 
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Figure 8. Sample RE model execution sensitivity analysis. 


10.0 Conclusions 

The RUNSAFE Bayesian Belief Network (BBN) model for Runway Incursion (RI) and 
Runway Excursion (RE) events has been presented. Numerous considerations surrounding the 
process of developing the RI and RE models have been documented in this report. The resulting 
RUNSAFE model (both RI and RE event models) has been thoroughly reviewed by a Subject 
Matter Expert (SME) panel through multiple SME knowledge elicitation sessions. Numerous 
improvements to the model structure (definitions, node names, node states and the connecting link 
topology) were made by the SME panel. The structural details of the resulting RUNSAFE BBN 
models for RI and RE events have been documented within this report. A few sample executions 
of the final RI and RE models, using random conditional probability tables have been presented 
for the baseline and worst-case scenarios; the resulting probability of an accident or near miss 
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increases substantially for the worst-case scenario, compared with the baseline scenario. Finally, 
a parameter sensitivity analysis for a given scenario was performed to show the risk drivers. 


11.0 Recommendations 

It is recommended that the model structures presented herein and the CPT values developed 
by the SME panel be validated by comparison to available data , be expanded to include the 
injection of technology products intended to improve runway safety, and that SME input be used 
to characterize the impact of these technology products. It is also recommended that the resulting 
BBN for RI and RE events be used by NASA to generically model the causes of RI and RE events 
and to assess the effectiveness of technology products being developed under NASA funding. 
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